Apparatus and method for augmenting sight

ABSTRACT

A method of augmenting sight in an individual. The method comprises obtaining an image of a scene using a camera carried by the individual; transmitting the obtained image to a processor carried by the individual; selecting an image modification to be applied to the image by the processor; operating upon the image to create a modified image using either analog or digital imaging techniques, and displaying the modified image on a display device worn by the individual. The invention also relates to an apparatus augmenting sight in an individual. The apparatus comprises a camera, carried by the individual, for obtaining an image of a scene viewed by the individual; a display carried by the individual; an image modification input device carried by the individual; and a processor, carried by the individual. The processor modifies the image and displays the modified image on the display carried by the individual.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/709,984 entitled “Apparatus and Method for Augmenting Sight” filedSep. 20, 2017, currently pending, which itself claims the benefit ofpriority as a continuation of U.S. patent application Ser. No.13/947,376 entitled “Apparatus and Method for Augmenting Sight” filedJul. 22, 2013, which itself is a continuation of U.S. patent applicationSer. No. 13/371,521 entitled “Apparatus and Method for Augmenting Sight”filed Feb. 13, 2012, which issued as U.S. Pat. No. 8,494,298 on 23 Jul.2013, which itself is a continuation of U.S. patent application Ser. No.12/060,964 entitled “Apparatus and Method for Augmenting Sight” filedApr. 2, 2008, which issued as U.S. Pat. No. 8,135,227 on 13 Mar. 2012,which claims priority from U.S. Provisional Patent Application60/921,468 entitled “Apparatus and Method for Augmenting Sight” filedApr. 2, 2007.

FIELD OF THE INVENTION

The invention relates generally to the field of vision care and morespecifically to the field of augmenting sight.

BACKGROUND OF THE INVENTION

There are conditions under which even individuals with 20/20 vision needsight augmentation. Such conditions may be brought on by low lightlevels, low or no color differentiation between objects in the visualfield, or the small size of the object viewed, to name but a few.Individuals with less than optimal vision or with other visual defects,such as retinopathies, also need augmentation to correct for theirvisual defects.

FIG. 1 is a schematic diagram of the eye. A portion of the retinaresponsible for fine detail vision is the macula. One form of visualdefect is AMD, or age-related macular degeneration. In maculardegeneration, which begins with the deposit of druesends in layersbeneath the photoreceptors, the degenerative process affects mostly themacula and results in death of cells necessary for vision. In somepatents, the result of macular degeneration is a central visual fieldblind-spot or scotoma. At this time there is no cure for AMD. Otherdiseases (such as, but not only, diabetic retinopathy, glaucoma, macularedema, optic nerve atrophy, etc.) can also result in significant lossesin vision, sometimes macular, sometimes peripheral, to this region ofhigh quality vision. Furthermore, the diseases' impact on vision isunique for each patient. What these vision losses have in common is theloss in quality of life due to the limited quality of vision.

There have been attempts to augment the quality of the perceived visualfield using such items as image intensity amplifiers, or “night scopes”,or mechanical or electrical image magnifiers. These devices tend to bebig, bulky, limited in their application and not appropriate fornon-military or paramilitary uses.

What is needed then is a general device that is capable of augmenting animage to be viewed by an individual, whatever the source of the image,be it a computer display, a television or other image source, under thecommand of that individual, to aid the individual in poor viewingconditions or to overcome physiological and psychological visual defectsin the individual. The present invention addresses this need.

SUMMARY OF THE INVENTION

The invention, in one aspect, relates to a method of augmenting an imageto be viewed by an individual. In one embodiment, the method comprisesthe steps of: obtaining an image of a scene viewed by the individualusing an image capture device carried by the individual; transmittingthe obtained image to a processor carried by the individual; selectingappropriate image modification to be applied to a region of interest in(ROI) in the image by the processor; and operating, by the processor,upon the image to create a modified image in response to the selectedimage modification or modifications; and displaying the modified imageon a display device worn by the individual. In one embodiment, the imagemodification is magnification. In another embodiment, the imagemodification is a remapping of the image to avoid visual defects in theindividual. In another embodiment, the image modification isminification, or fractional magnification. In another embodiment, theimage modification overlays minified peripheral information into thecentral area of the field of view. In yet another embodiment, the imagemodification is a remapping of colors within the image. In still yetanother embodiment, the image modification is edge enhancement. Inanother embodiment, the image modification is image intensityenhancement. In one embodiment, the image modification takes placesubstantially in real time. Other embodiments may include combinationsof these and other functions.

In another aspect, the invention relates to an apparatus for augmentingan image viewed by an individual. In one embodiment, the apparatuscomprises an image capture device carried by the individual, forobtaining an image of a scene viewed by the individual; a displaycarried by the individual; an image modification input device carried bythe individual; and a processor carried by the individual. The processoris in communication with the image capture device, image modificationinput device and display. The processor modifies the image obtained bythe image capture device, in response to the instructions provided bythe individual using the image modification input device, and displaysthe modified image on the display carried by the individual.

In one embodiment, the display comprises a pair of glasses. In anotherembodiment, the image capture device is attached to a pair of glasses.In yet another embodiment, a second image capture device and a seconddisplay are in communication with the processor. The second imagecapture device provides a second image for the processor to modify anddisplay on either the first or the second display.

In yet another aspect, the invention relates to a method for improvingthe visual function. In one embodiment, the method includes the stepsof: determining the locations of retinal damage in an eye of thepatient; obtaining an image of a scene to be viewed by the patient; andmapping the image to a display in such a way to avoid the locations ofretinal damage when the display is viewed by the patient. In anotherembodiment, the step of obtaining the image uses an image capture deviceattached to the glasses of the patient. In yet another embodiment, thedisplay replaces a lens of the glasses of the patient.

Another aspect of the invention relates to an apparatus for improvingthe visual acuity of a patient with a degenerative disease of theretina. In one embodiment, the apparatus includes a camera for obtainingan image of a scene to be viewed by the patient; a display; a memorystoring locations of retinal damage of the eye of the patient; and aprocessor, in communication with the image capture device, display andmemory, for mapping the image to the display in such a way as to avoidthe locations of retinal damage when the display is viewed by thepatient. In another embodiment, the display replaces a lens of theglasses of the patient.

DESCRIPTION OF THE DRAWINGS

The invention is pointed out with particularity in the appended claims.The advantages of the invention described above, together with furtheradvantages, may be better understood by referring to the followingdescription taken in conjunction with the accompanying drawings. In thedrawings, like reference characters generally refer to the same partsthroughout the different views. The drawings are not necessarily toscale, emphasis instead generally being placed upon illustrating theprinciples of the invention.

FIG. 1 is a diagram of the eye;

FIG. 2 is a highly schematic diagram of an embodiment of the system ofthe invention;

FIG. 2a is a flow diagram of an embodiment of a method, implemented bythe system of FIG. 2, to modify an image;

FIG. 2b is an example of an image modified by this method;

FIG. 3a is an example of an image as viewed with a blind spot defect inthe visual field;

FIG. 3b is an example of an image as viewed with a blind spot defect inthe visual field but with the image magnified;

FIG. 3c is an example of an image as viewed with a blind spot defect inthe visual field but with magnification and deformation generated by a“pushout” algorithm;

FIG. 3d is an example of an image as viewed with a blind spot defect inthe visual field but with magnification and horizontal spacing generatedby a “horizontal split” algorithm;

FIG. 4(a-c) respectively depict an image, the image with the gradientapplied and the image with the gradient applied with suppression;

FIG. 5 is a flow diagram of an embodiment of a color-mapping algorithm;

FIG. 6(a-c) depicts a grayscale rendering of the results of mappingcolors as would be seen by someone with red-green color blindness.

DETAILED DESCRIPTION

In brief overview and referring to FIG. 2, the system for augmentingsight in one embodiment includes a pair of eyeglass frames 10 orheadmounted display, such as an Nvisor SX, by NVIS (Reston, Va.), and aprocessor 14. In one embodiment, the processor 14 is a general purposecomputer, such as made by Shuttle Computer Group (City of Industry,Calif.). The eyeglass frames 10 are the typical eyeglass framesgenerally available and used today with transparent lenses. In thisembodiment, the transparent lenses have been replaced with one or twodisplay screens 18, 18′ (generally 18). Attached to the frame are one ormore image capture devices 26, such as a camera. In one embodiment, theimage capture device is a Microsoft 2.0M Webcam (Redmond, Wash.).Optionally, one or more eye or pupil tracking sensors 28 and associatedelectronics are also attached to the frame. The electronics provide forimage capture by the image capture device and transmission to theprocessor 14 by way of a wired or wireless link 50. The processor 14includes one or more input output (I/O) modules 34, 34′, 34″ and amemory 38 in communication with each other by way of a bus as instandard computer design. The I/O modules 34, 34′, 34″ not only receiveimages from the image capture device 26, but transmit the modifiedimages back to the eyeglass frames for display on one or both of thedisplay screens 18, 18′. With two or more image capture devices 26, theresulting images each may be displayed on a respective display 18, 18′to provide depth perception (depending on the capture device position),or one image capture device 18 can select an region of interest (ROI) inthe field of view (FOV) of the other image capture device 18′ anddisplay the region of interest within the field of view on bothdisplays. In this way, for example, a magnified region of interest maybe displayed in the larger field of view.

In more detail, in various embodiments, the displays 18, 18′ in theeyeglass frames 10 include, in one embodiment, a thin film display suchas a liquid crystal display. In another embodiment, the displays useLiquid Crystal on Silicon (LCOS) technology. In a further embodiment,the displays use Organic Light Emitting Diode (OLED) technology. Instill a further embodiment, the displays use micro-projection technologyonto a reflective (partial or 100% reflective) glass lens. In variousembodiments, each display shows a different image or the same image. Ifthe modified image is to be displayed only to one eye, only one display18 is required. The displays in various embodiments can incorporaterefractive lenses similar to traditional eyeglasses, such that thedisplay works in concert with a person's unique optical prescription.

Similarly, the image capture device 26 in one embodiment is a chargecoupled device (CCD) camera with high depth-of-field optics. In anotherembodiment, the image capture device is a Complimentary Metal OxideSemiconductor (CMOS) image sensor with appropriate optics. In othervarious embodiments, the image capture device is any imaging device withan analog or digital signal output that can be sent to a processing unit14 for processing. In a binocular configuration, each image capturedevice or camera 26 sees a slightly different image, thereby providingstereoscopic vision to the viewer. If the image is to be presented toonly one eye, then only one image capture device or camera 26 is neededto record the image for that eye. Although in the embodiment shown theimage capture device or camera 26 and related electronics are mounted onthe eyeglass frames 22, it is contemplated that the camera 26 andelectronics could also be located elsewhere on the individual's person.Also, although two cameras 26 are contemplated for binocular vision, itis possible for one camera 26 to view the image and present the sameimage to both displays 18. In addition, in various other embodiments thesource of the image may be another camera, a television, a computer 54or other source 58 capable of supplying an input to the processor 14.

The optional eye tracking sensor 28 is also in communication with theelectronics and determines where in the visual field the individual islooking. In one embodiment, this sensor 28 operates by following theposition of the pupil. Such eye tracking devices 28 are common inpresently available “heads-up-displays” utilized by military pilots.Again, although an embodiment contemplated includes two tracking sensors28, because both eyes typically track together, one tracking device maybe used. In another embodiment, the eye tracking sensor uses acombination of mirrors and prisms such that the optical path for the eyetracking sensor is orthogonal to the pupil. Eye tracking is used todetermine the region of interest (ROI), and to ensure that the damagedareas of a person's vision are avoided when the modified image ispresented to the eye. The eye-tracking information is suitably averagedand dampened in software to minimize the sensitivity to random eyemovements, blinks, etc., and to optimize the system for various usagemodels. For example, reading English requires specific eye trackingperformance in the left to right direction different from that in theright to left direction, and different again from that in the verticaldirection.

Images from the image capture device 26, eye position information fromthe eye tracking device 28 and images destined for the displays 18 arepassed through the appropriate I/O module 34, 34′, 34″ (HDMI to VGA, PCIand USB respectively) of the processor 14. In the embodiment shown, thedisplay on the NVISOR SX display unit is controlled by an nVIScontroller 52 by the same manufacturer of the NVISOR SX display 18. Thiscommunication between the processor 14 and the electronics of theeyeglass frames 10 may be transmitted through a wired connection 50 orbe transmitted wirelessly. Certain functions, such as magnification, maybe performed in an analog manner, such as by adjusting the lens array onthe camera or digitally by mathematically processing pixels.

In the embodiment shown, the processor 14 is a Shuttle computer havingmemory 38 and I/O modules 34, 34′, and 34″. The I/O modules not onlycommunicate with the eyeglass frames 10 but also with other displays andinput devices. For example, the processor 14 may be connected to asecond optional monitor 46, so that a health care provider or devicetechnician can see what the wearer is seeing. In addition, the NVIScontroller 52 is capable of providing video data to a projector 56. Inthis way, greater numbers of individuals may see what the wearer isseeing.

Additionally, display images from a computer 54 and from a video source58 such as a DVD may provide images for display on the display of theeyeglass frames 10. Such images may be used to help train the wearer todiagnose hardware and software failures and to help diagnose and treatthe patient. In one embodiment, an input device such as a DVD player 58provides a signal to an RF modulator 62 which then passes the RF imagesignal to the processor 14 through a Win TV NTSC to USB module 66. Thissignal enters the processor 14 through a USB connector 34″. Similarly,image data from a computer monitor 54 may also be displayed on theglasses 10 by converting the signal from the monitor 54 using a VGA toUSB converter (for example an Epiphan Systems converter, Ottawa,Ontario, Canada.) 68. Additionally, the user may wear a ring-like“text-camera” on his or her finger which he or she then scans over aline of text. Such devices reduce the optical complexity of the eyeglasscamera 26. Finally, in this embodiment, input commands may be entered byway of a microphone 48 in communication with an iPAQ computer 66.

The processor 14 in another embodiment is a processing device havingcellular telephone capabilities or a software modified cellulartelephone. In this embodiment data, for example from an ophthalmologistor other health care professional 46, may be received from the cellulartelephone network and verbal control instructions from the individual 48may be input through the phone's microphone or alternatively may bekeyed in through the phone's touchpad or movement sensor. In otherembodiments, the processor is a specialized computer or handheld device.

Received data and control instructions are then stored in memory 38. Thememory 38 includes random access memory (RAM) for data storage andprogram execution, and read only memory (ROM) for program storage. Themicroprocessor 14 accesses the data in memory and manipulates it inresponse to the control instructions for transmission back to theeyeglass frames 10 for display. In this way, the individual can tailorthe displayed image for optimal viewing.

One embodiment of the method using the system which is capable ofmodifying an image of the field of view is shown in FIG. 2a . The wearerbegins by setting the preferred method of determining the location ofthe region of interest (ROI) through a keyboard or other input device(step 10). The individual may indicate their preferred location of theROI by selecting one of a mouse input (step 12), preset coordinates(step 14) or eye-tracking imaging (step 16).

If an eye tracking sensor 28 is used, the individual need only movetheir eye to determine the region of interest (step 18). Somemathematical parameters are applied to determine the sensitivity of theeye tracking algorithm in the X and Y directions (step 20) to minimizethe effect of involuntary eye movement on the choice of region ofinterest.

From this information, the center of the region of interest (ROI) isdetermined. If the region of interest (ROI) is not within the viewingarea (step 22), the region of interest is set to the last valid regionof interest (step 24). The complete region of interest (ROI) is thendetermined, or “mapped” such that it is centered on the coordinatesdetermined (step 26). The size and shape of the ROI is determinedthrough user inputs (step 28).

The visual information in the region of interest (ROI) may be input fromeither the field of view (FOV) image (step 32), or from a separateregion of interest image source (step 34), as determined by user input(step 36). If the ROI image is to come from a separate source (step 36),then the user can input an optical zoom requirement (step 38) for thisimage. The ROI image is then captured (step 40) and overlaid or mapped,onto the ROI area (step 42).

The individual sets the zoom requirement (step 44) for the field of view(FOV) image. The zoom function is a combination of both optical zoomdone in the FOV camera using lenses, and digital zoom performed insoftware. The FOV image is then captured. (step 44).

The image is then modified (steps 24 and 25) as further required by theuser input values (steps 46 48, and 54). Note that some modificationsare applied to the left and right displays, or left and right eyes,differently (step 52), while others are applied to the left and rightdisplays equally (step 54). Any of the image modifications may beapplied to either the region of interest (ROI) or the entire field ofview (FOV), or both. The final modified images are then presented to thedisplays (step 58). FIG. 2b depicts what the displayed magnified textwould look like.

Referring also to FIG. 3, the system can also be used to correct visiondefects in the eyes of the individual. In this example, an individualhas a defect in his or her visual field that causes a perceived imagedefect as shown in FIG. 3a . As a first step, an ophthalmologistperforms an eye examination on the individual, mapping the areas of theeye which are not functioning properly. This information is downloadedto the memory 38 of the processor 14 through the I/O module 34. Theprocessor can then map the image to avoid the defect as is shown inFIGS. 3b, 3c and 3d . The end result is that the remapped image removesloss of information (previously hidden behind the defect) caused by thedefect as shown in FIGS. 3b and 3c . In FIG. 3b the text is magnifiedabout the defect region, while in FIGS. 3c and 3d the text is remappedto be spaced about the defect. Thus, with training the individual iscapable of seeing a full image substantially free of distortion. Theindividual may perform many types of image modification by entering datathrough the keypad of the device or by speaking instructions through themicrophone of the device.

The device is designed to help anyone having to deal with visualchallenges which cannot be addressed by simple optical means (glasses,contact lenses, etc). Visual challenges can be due to either less thanoptimal performance of the visual system or environmental conditions.The visual system is a complex structure which combines an opticalimaging system (the front end of the eye), a network of sensors (thephotoreceptors) positioned at or near the focal plane of the imagingsystem and a complex neural network (and its supporting infrastructureof cells) for processing the information from the sensors into a visualsignal. A problem in either the optical, sensing or neural component ofvision will result in less than optimal vision. The resulting visualproblems can manifest themselves in many ways including, but not limitedto, a reduced ability to see fine details; a reduced sensitivity tocontrast; a reduced ability to extract color information; a loss inperipheral field of view; a loss of central field of view; and anincreased sensitivity to brightness.

These various types of vision loss can be the result of trauma to theeye or disease of the eye. Most of these diseases affect the back of theeye (retina) where light sensing and some signal processing occurs.Glaucoma, diabetic retinopathy, age-related macular degeneration (AMD),and retinitis pigmentosa are some of the more common causes of visionloss in the developed world. The resulting visual problems and theirextent vary from almost no noticeable effect to complete blindness andare unique to each patient.

The invention is not disease specific, and is able to address the majordiseases discussed above as well as most other retinal conditions (suchas, but not limited to retinopathies, optic disc neuropathies,Stargardt's disease, retinal dystrophies, most variations ofmacular/foveal edema, etc.) short of profound blindness, by dramaticallyimproving the wearer's visual experience and ability to function beyondthat which is possible without the invention.

The proposed solution can also be helpful, if likely to a lesser extent,to patients with degraded optical properties including optical errors inthe cornea (front element of the eye), the crystalline lens (lens insidethe eye) and any issues with the liquid contained within the eye(scattering sites, opacification, etc.).

Finally, the invention can also help some people with visual problemsdue to higher level processing errors in the brain such as, but notlimited to, compensating for missing portions of their field of view,problems with tracking, problems that are helped by improving mentalfocus and removing peripheral distractions (such as dyslexia), etc.

Outside of visual problems, there are many environmental conditions thatcan lead to poor visual information transfer. As an example, trying tolook at someone's face while they stand in front of a window on a brightsunny day, looking at a baseball game where part of the field is insunlight and another in shadows, poor quality illumination (lots of bluefor example). The device can certainly help most of these people reducethe impact of the environmental condition on their visual performance.These conditions can occur during work or leisure activities, forexample facing the sun up on a telephone pole while performing a repair,walking the dog, attending a sports event, etc.

Finally, the device can enhance the amount of information available tonormally sighted people. It can overlay multiple sources of informationon a same field of view. This can be used in professional applications,for example, to call up stock figures or inform a wearer of incomingemail overlaid upon a real-world image while walking down the street; tocall up an electrical wiring diagram overlaid with a magnified image ofbroken down electric circuit to effect a repair. These images will notonly be overlaid, but can be manipulated to optimize informationdelivery and minimize disturbance from natural visual experience. Also,the invention enables hands-free access to this information, which iscritically important in some applications.

To correct for these conditions the user can issue instructions thatcause the processor 14 to perform operations on the image including butnot limited to:

1. Magnify field of view (FOV) or ROI—this function permits the field ofview to be decreased and the resolution increased up to the resolutionof the camera and the resolution of the display.

2. Minification: Reducing the FOV to a smaller size to account forconditions which manifest themselves as “tunnel vision”. This isequivalent to fractional magnification.

3. Enhance contrast in entire FOV or only ROI—this function permitscontrast contained naturally in the image to be modified so as toenhance the difference between various levels of contrast to improve thedetection of information in the image.

4. Enhance edges in entire FOV or only in ROI—this function permits theedge of an object in the field of view to be detected and enhanced (forexample, but not limited to, adding a black band) to improve the abilityof the patient to perceive the edges of different features of the image.

5. Change to grey scale in entire FOV or only in ROI—this functionpermits the image to be converted to a grey scale from a color scale.

6. Threshold grey scale in entire FOV or only in ROI—this functionpermits all the colors and intensities of the image to be mapped intoeither black or white.

7. Remap colors in entire FOV or only in ROI—this function remaps thecolors in the original image into another range of colors, therebypermitting color blindness or deficiency to be ameliorated.

8. Remap image based on the user's blind spot in ROI—this functionallows the individual to remap the image to avoid the blind spots causedby diseased regions of the eye, such as in macular degeneration orStargardt's disease. Various algorithms relocate pixels from behind ablind spot to areas near the periphery of the blind spot according to amathematical spatial distribution model.

9. Relocation and Enhancement of Text: This technique is a specificimplementation of “Spatial Remapping” above, where text is moved outfrom behind a blind spot. The technique includes application sensitivetechniques such as only splitting the image on the blank lines betweentext lines, serif removal, text edge smoothing, text enhancement throughcolor and contrast improvement, optical character recognition (OCR),etc.

10. Brightness adjustment of field of view or region of interest:Individual pixels can be modified to increase or decrease theirbrightness either globally or according to a mathematically definedspatial distribution.

11. Brightness flattening of field of view or region of interest: Thevariation in brightness across an image can be reduced, such that“hotspots” or washed out regions are darkened, and dark areas arebrightened.

12. Image Superimpositioning: This is a technique where peripheralinformation is overlaid into a central area of the FOV, in order toprovide contextual data to people with lost peripheral visualperformance.

14. Color Identification: The invention can identify (via screen text)the dominant color or the statistical red-green-blue (RGB) content for aspecific portion of the image, as identified for example by“cross-hairs.”

15. Black/White Conversion and Inversion of field of view or region ofinterest: Color or grayscale images can be reduced to B/W or invertedB/W (W/B).

By using fast processors it is possible to make these modifications insubstantially real time. This allows a visually impaired individual tofunction substantially as if there were no visual defect. With a fastenough computer, these enhancements may be applied and removedsequentially to an image, that is the image toggled between the actualimage or the image as modified, by the user so that the user sees theoriginal image and the enhanced image as a repeating toggled sequence.This provides the user with a clearer sense about what aspects of thepresented image are “real” and which are “enhancements”.

Further certain enhancements can be applied and removed from the imageautomatically. For example, an edge enhancement modification can beapplied and removed sequentially and repetitively such that the usersees an edge enhanced image and then the unmodified image.

Many algorithms can be used to achieve these purposes. For example, oneembodiment of an edge finding algorithm detects edges using a gradientoperator. To avoid noise due to small natural variations in intensity ofthe image, the gradient operator is applied to a low pass digitallyfiltered version of the image. If the digital filter is a Gaussian, thenthe gradient of the filtered image is simply the convolution of theimage with the gradient of the filter; the Canny Gradient Operator. Thistechnique has two major advantages. Firstly, this technique avoids theissue of having to calculate a finite derivative of the natural image.Although the derivative of the Gaussian function is known analytically,the derivative of the natural image is mathematically ill-posed. Second,this technique permits both the filtering and derivative operations to)be performed simultaneously in Fourier space. This is represented by:∇ƒ_(σ)(x,y)=(ƒ*∇g _(σ))(x,y)

where ƒ and ƒ_(σ) are the unfiltered and filtered images respectivelyand g_(σ) is the Gaussian filter. The amount of filtering applied willbe controlled by the Gaussian width (σ). One embodiment of theimplementation separates the gradient operator into its two Cartesiancoordinates, so that in its final form the gradient is:

${{\nabla_{x}{f_{\sigma}\left( {x,y} \right)}} = {\left( {f*\frac{\partial g_{\sigma}}{\partial x}} \right)\left( {x,y} \right)}},{{\nabla_{y}{f_{\sigma}\left( {x,y} \right)}} = {\left( {f*\frac{\partial g_{\sigma}}{\partial y}} \right)\left( {x,y} \right)}},{{M_{\sigma}\left( {x,y} \right)} = \sqrt{\left( {\nabla_{x}{f_{\sigma}\left( {x,y} \right)}} \right)^{2} + \left( {\nabla_{y}{f_{\sigma}\left( {x,y} \right)}} \right)^{2}}},{{\Theta_{\sigma}\left( {x,y} \right)} = {a\mspace{14mu}{\tan\left( \frac{\nabla_{y}{f_{\sigma}\left( {x,y} \right)}}{\nabla_{x}{f_{\sigma}\left( {x,y} \right)}} \right)}}}$

This generates an amplitude term (M) which is the vector sum of the twocomponents and a direction component (Θ). The result of this filteringis a gradient map which does not show edges specifically. The gradientimage is then processed to identify edges by first using a bi-linearinterpolation around each point in the image to identify the pointswhich are local maxima. Once identified, only the local maxima areretained and all other points are ignored. Then the direction of thegradient is used to identify adjacent points which are connected,because the gradient will be similar for adjacent points if they arepart of the same edge. Other outliers in the gradient are rejected.Finally, a thresholding algorithm is applied which retains all gradientpoints having a value in the upper percentile (in one embodiment,threshold 1, the 90^(th)) and rejects all weak gradients having a valuein the lower percentile (in one embodiment, threshold 2, the lowest20^(th)). Anything in between the two thresholds is rejected if it hasno strong companion near it, and kept if its neighborhood indicates anedge. All retained gradient points are then binarised to 1, all othersto 0, creating the outline of edges in the image. FIG. 4a depicts animage in its natural state. FIG. 4b depicts the image of FIG. 4a with agradient applied, and FIG. 4c depicts the image of FIG. 4b withsuppression of the underlying image.

Similarly, an example of a color remapping algorithm is next described.Normally sighted people depend on both brightness and color differences(luminance and color contrast) to identify features in their visualfield. Abnormal color vision will often result in the inability todistinguish between colors; a reduced capacity to use color contrast toextract information. Color confusion is usually asymmetric, so thatcolor confusion occurs along the Red-Green or Yellow-Blue color axis.This means that by remapping colors in the field of view which areconfusing to an observer to color in the spectrum which offer bettercontrast, it is possible for the user to recover the information contentof the field of view.

The algorithm described below is intended to remap the color containedin the field of view to allow the user to extract maximum contentinformation. The color content of the processed field of view will notbe true to the real world thus actual color information will not alwaysbe natural, but the color contrast will be enhanced for the observer sothat there will be little or no confusion due to reduced color contrastbetween the objects in the field of view. This will allow the observerto identify a maximum number of details and maximize informationextraction.

The algorithm is illustrated in FIG. 5. If a color perception defect isidentified in a patient, then the image is modified by shifting some ofthe color in the defective color channel (Red-Green or Blue-Yellow) inthe other color channel. Two parameters are typically required. Thefirst is to identify which colors in the image must be modified, and thesecond is to determine the amplitude of the color shift necessary tomove the affected colors to the unaffected color channel.

First, the colors to be modified are selected by the amount of theaffected primary color (Red, Green or Blue) in the image. For example,if the color defect is the inability to detect color contrast in thered/green channel, then either the reds or greens are shifted to theblue channel; whichever gives the observer the best contrast. Given thatWhite will contain 33% of each Red, Blue and Green primary color, thenthe threshold for shifting a given primary color should be >33%. Thethreshold will be both observer and image dependent and will need to beadjustable. The amount of remapping to the better color channel willalso be observer dependent as well as image dependent and thus it toowill also need to be adjustable.

For each point in the image, where R, G and B represents the intensityof each primary color, the algorithm proceeds as follows:

First, the RGB values are measured, and the brightness (T) (T=R+G+B) andthe normalized color values (r, g, b) (r=R/T, g=G/T, and b=B/T)calculated. Next, for each point in the image where the color containsmore than the threshold amount of the problematic primary color, apercentage, shf, of the problem primary is shifted into another primarycolor.

For example, if (r) is the normalized value of the problematic colorthen:

If r>0.4 then red the primary color is more than 40% of the color of theimage and hence above the threshold.

r(n)=(1−shƒ(r)), where r is the normalized value of the problematiccolor, and r(n) is the new normalized value for the shifted red primarycolor. Similarly, b(n)=b+shƒ*r where b(n) is the new normalized valuefor blue primary. Finally, g(n)=g which means the normalized primarycolor green (g) is unmodified.

One skilled in the art would recognize that if red is not theproblematic color, then similar shifts are possible for the otherprimary colors. Thus, if the problem primary color is green (g) then thealgorithm will shift some of the primary green color (g) into blue.Similarly, if the primary color blue is the problem, then the algorithmwill shift blue into red.

The new RGB coordinates of the point being examined is then the newnormalized shifted color times the brightness T. Thus Rn=rn*T, Gn=gn*Tand Bn=bn*T. The results of this algorithm are shown in FIGS. 6a -c.

An embodiment of the algorithm for automatic brightness and contrastenhancement transforms the image based on the intensity (signal)histogram distribution for the whole image. This technique is usuallyreferred to as brightness/contrast equalization. An intensitydistribution (number of pixels at each intensity levels), D_(A), fromthe original image (A) is remapped into a new image (B) withdistribution, D_(B), with the constraints that the remapping result besingle valued (each intensity level in D_(A) can only transform to asingle intensity level in D_(B)) and that the transform be reversible ormonotonic.

These constraints are embodied in the equations:D _(B)=ƒ(D _(A))andD _(A)=ƒ⁻¹(D _(B))

Many different transforms can be used that meet these constraints. Oneembodiment is the algorithm discussed below. This algorithm is a simpleand effective approach that is widely used in the image processingworld.

This embodiment of the algorithm adds additional constraints to thedetermining the mapping function ƒ(D_(A)). In one embodiment, anadditional requirement is that the energy contained within a smallregion (dD_(A)) of the distribution D_(A) must equal the energy to thecorresponding region dD_(B) of the distribution D_(B). That is:h _(A) *dD _(A) =h _(B) *dD _(B)

where h is the number of pixels at a predetermined intensity level, (x).If the values of h are rescaled by dividing the value by the totalnumber of pixels then the values of h can be expressed as probabilitydistributions p_(A) and p_(B). Furthermore, because the intensitydistribution is being stretched from the original image (0 to a maximumintensity, D_(M)) and because the area under the two probabilitydistributions must be equal as described above, then the derivative ofthe transfer function dƒ=dƒ(x)/dx, can be set to a constant equal toD_(M). The transform function is then rewritten in terms of theprobability distribution p_(A) and D_(M):ƒ(D _(A))=D _(M) *∫p _(a)(u)du=D _(M) *F _(A)(D _(A))

where F_(A)(D_(A)) is the cumulative distribution function for theoriginal image. The implementation then becomes:

First, obtain an intensity distribution function for the original imagewith the same number of bins available as there are available greylevels for the display mode (that is, 8 bits gives you 256 potentialbins.)

Next, normalize the distribution function by dividing it by the numberof pixels to convert the distribution function to a probabilityfunction.

Third, find the largest gray level with a non zero value in the originalimage and set this to D_(M).

Next create a cumulative distribution function: For example bin 0 is thenumber of pixels of brightness=0; bin 1 is sum of the number of pixelsin bin 0 and 1; bin 2 is sum of pixels in bins 0, 1, 2; and so on.

Fifth, for each pixel, obtain the intensity, I(c,r) where c and r arethe column and row indices, and find the cumulative probability for thatintensity I(c,r); a value between 0 and 1.

Then multiply this value by D_(M). This is the new value of theintensity for that pixel, after equalization.

Finally, to obtain stretching as well, multiply the new intensity valueby the ratio of the maximum possible for display divided by D_(M). Thisstep ensures the maximum contrast. FIG. 6 shows a grey-scale image of acolor blindness test image. FIGS. 6b and 6d depicts grey-scale images ofthe color blindness test image with the green shift to blue and redshifted to blue, respectively. Thus a person with red-green colorblindness would be able to easily see portions of the image which wouldnormally appear hidden.

A patient may use any function which addresses his or her visual defectsby entering the requested function using the keypad. However, theparameters which the system used to correct for the defects may need tochange over time. This is because typically, over time, the patient'svisual preferences may evolve; or the visual defect may worsen due tothe aging process, due to an accident, or disease. Further, a patientmay simply prefer to change the configuration settings based on thecurrent task they are performing, and as such may have differentpreferences for different tasks. Thus a user, using the control featuresfor the display system, can adjust the settings of the optical display,allowing the user to make minor changes to his or her prescription.

When a user requires a minor change to his or her vision systemsettings, he or she can either go to a vision care professional, whowill change the system settings, or change the settings themselves. Forexample, the user is able to configure a ‘recipe’ of image modificationsoftware algorithms to correct or enhance his or her vision in a simple,time-efficient way for a defined set of visual tasks such as watchingTV, reading, playing bridge, needlepoint, etc. without the assistance ofa specially trained clinician.

For major changes to the system settings, professional ophthalmicoversight may still be required. For example, the ability for aclinician to synthesize, review, modify and, if deemed appropriate,approve a user-selected image enhancement ‘recipe’ as above, may berequired for regulated activities such as driving. The software ‘recipe’would not become operational in the system, which is identified by aunique software serial number, unless and until activated by theclinician. Typically, the clinician is also identified by a uniquesecure government identification number provided to those cliniciansauthorized to approve visual aids for driving. The clinician mayinteract with the system directly or may remotely connect to the system.Upon clinician approval of the prescription, the clinician would thenreceive compensation for services. The compensation is provided by wayof funds transfer from one or both of the system manufacturer,distributors or dealers and the user or his or her insurance company.The funds transfer in one embodiment is done electronically.

For example, in one embodiment, the user would enter the changesrequired to their settings to a settings application running on thesystem. The requested changes would then be indicated to the user'sclinician or eye doctor by way of the settings application, allowing thenew prescription to be downloaded by the optometrist or ophthalmologist.The optometrist or ophthalmologist's office system would first be paidfor services rendered, by system supplier directly for initially sellingthe system. All fees for ‘Optometric oversight’ functions or adjustmentswould be paid directly by the insurance company or individual patient,to the eye care professional or clinician. The system adjustments couldalso be used to make changes to the user's prescription such that a‘configurable low vision aid’ version of the system which incorporatesmulti-diopter lens characteristics could be used instead of refractivelenses.

While the present invention has been described in terms of certainexemplary preferred embodiments, it will be readily understood andappreciated by one of ordinary skill in the art that it is not solimited, and that many additions, deletions and modifications to thepreferred embodiments may be made within the scope of the invention ashereinafter claimed. Accordingly, the scope of the invention is limitedonly by the scope of the appended claims.

What is claimed is:
 1. A method of improving the visual function of auser comprising the steps of: configuring within a non-volatilenon-transitory memory associated with an electronic processor of a firstelectronic device a plurality of sets of second data; each set of seconddata relating to configuring a mode of operation of a plurality of modesof operation and relates to a predetermined subset of a plurality ofimage processing algorithms applied by the electronic processor;obtaining image data from a second electronic device; modifying apredetermined portion of the obtained image data with the electronicprocessor to generate modified image data established in dependence uponapplying the predetermined subset of the plurality of image processingalgorithms associated with a selected mode of operation of the pluralityof modes of operation to the predetermined portion of the obtained imagedata; projecting an image comprising the modified image data onto auser's retina via a display; wherein the selected mode of operation isestablished by the electronic processor either: by receiving input dataestablished by the user via an input interface of a portable electronicdevice coupled to the first electronic device, the input data relatingto the selection by the user of a mode of operation of the plurality ofmodes of operation; or in dependence upon the visual field of view ofthe user; user generated adjustments to at least one of configure andrevise the mode of operation of the plurality of modes of operation upto a predetermined level of change are employed by the first electronicdevice without professional oversight; and above the predetermined levelof change professional oversight is required to approve the useradjustments.
 2. The method according to claim 1, wherein at least oneof: the image comprising the modified image data comprises the obtainedimage data within which the predetermined portion of the image data hasbeen digitally replaced with the modified image data; and thepredetermined portion of the obtained image data relates to apredetermined portion of a visual field of view of the user associatedwith a vision loss of the user.
 3. The method according to claim 1,further comprising either: storing within a non-volatile non-transitorymemory associated with an electronic processor of a third electronicdevice first data relating to a determined vision loss relating todegradation in a predetermined portion of a visual field of view of theuser; providing to the electronic processor of the first device thefirst data; and establishing the predetermined portion of the obtainedimage data in dependence upon the first data; or storing within anon-volatile non-transitory memory associated with an electronicprocessor of a third electronic device first data relating to adetermined vision loss relating to degradation in a predeterminedportion of a visual field of view of the user; providing to theelectronic processor of the first device the first data; andestablishing the plurality of modes of operation in dependence upon thefirst data.
 4. The method according to claim 1, wherein either: thedisplay forms part of a head mounted device comprising the firstelectronic device; or the display forms part of the head mounted devicewhich also comprises an interface supporting communications to and fromthe first electronic device.
 5. The method according to claim 1, whereinthe image comprising the modified image data comprises modified obtainedimage data within which the predetermined portion of the obtained imagedata has been digitally replaced with the modified image data; thepredetermined portion of the obtained image data is modified in order toaddress a first vision loss of the user with respect to a predeterminedportion of their visual field of view; and the modified obtained imagedata comprises the obtained image data modified in dependence upon asecond vision loss relating to a general degradation in the user'svisual field of view.
 6. The method according to claim 1, wherein atleast one of: the second electronic device is at least one of a digitalimage acquisition device, a source of video data being displayed uponanother display within the field of view of the user, and a thirdelectronic device associated with the user; and the predeterminedportion of the obtained image data to be modified is determined independence upon data acquired tracking the user's eye movement and thefirst data.
 7. The method according to claim 1, wherein thepredetermined portion of the obtained image data modified relates to acharacteristic of a vision loss of the user within a predeterminedportion of the user's visual field of view; the characteristic of thevision loss of the user relates to at least one of: an element theuser's optical imaging system selected from the group comprising cornea,pupil, lens, iris, and vitreous gel; a neural component of the user'svision; an environmental condition; compensating for missing portions ofthe user's field of view; correcting a problem with the user's eyetracking; improving mental focus; and removing peripheral distractions.8. The method according to claim 3, wherein the first data is generatedby at least one of an ophthalmologist and a healthcare professional. 9.The method according to claim 1, wherein each mode of operation of theplurality of modes of operation relates to a different task performed bythe user; and the predetermined subset of a plurality of imageprocessing algorithms associated with each mode of operation of theplurality of modes of operation is established by the user.
 10. A methodof improving the visual function of a user comprising the steps of:configuring within a non-volatile non-transitory memory associated withan electronic processor of a first electronic device a plurality of setsof second data; each set of second data relating to configuring a modeof operation of a plurality of modes of operation and relates to apredetermined subset of a plurality of image processing algorithmsapplied by the electronic processor; obtaining image data from a secondelectronic device; modifying a predetermined portion of the obtainedimage data with the electronic processor to generate modified image dataestablished in dependence upon applying the predetermined subset of theplurality of image processing algorithms associated with a selected modeof operation of the plurality of modes of operation to the predeterminedportion of the obtained image data; projecting an image comprising themodified image data onto a user's retina via a display; wherein theselected mode of operation is established by the electronic processoreither: by receiving input data established by the user via an inputinterface of a portable electronic device coupled to the firstelectronic device, the input data relating to the selection by the userof a mode of operation of the plurality of modes of operation; or independence upon the visual field of view of the user; user generatedadjustments to at least one of configure and revise the mode ofoperation of the plurality of modes of operation up to a predeterminedlevel of change can be employed by the first electronic device withoutapproval; and user generated adjustments to at least one of configureand revise the mode of operation of the plurality of modes of operationabove the predetermined level of change cannot be employed by the firstelectronic device without approval.
 11. The method according to claim10, further comprising establishing approval of the user generatedadjustments above the predetermined level of change by: transmitting theuser generated adjustments to a third electronic device associated withan individual providing professional oversight; review of the usergenerated adjustments upon the third electronic device by the individualproviding professional oversight; and transmitting to the firstelectronic device from the third electronic device oversight datarelating to approval or rejection of the user generated adjustments bythe individual providing professional oversight; and upon determiningwith the first electronic device that the oversight data relates toapproval of the user generated adjustments enabling the user generatedadjustments.
 12. The method according to claim 10, wherein approval isprovided by an ophthalmologist or an ophthalmic clinician.
 13. Themethod according to claim 10, wherein at least one of: the imagecomprising the modified image data comprises the obtained image datawithin which the predetermined portion of the image data has beendigitally replaced with the modified image data; and the predeterminedportion of the obtained image data relates to a predetermined portion ofa visual field of view of the user associated with a vision loss of theuser.
 14. The method according to claim 10, further comprising either:storing within a non-volatile non-transitory memory associated with anelectronic processor of a third electronic device first data relating toa determined vision loss relating to degradation in a predeterminedportion of a visual field of view of the user; providing to theelectronic processor of the first device the first data; andestablishing the predetermined portion of the obtained image data independence upon the first data; or storing within a non-volatilenon-transitory memory associated with an electronic processor of a thirdelectronic device first data relating to a determined vision lossrelating to degradation in a predetermined portion of a visual field ofview of the user; providing to the electronic processor of the firstdevice the first data; and establishing the plurality of modes ofoperation in dependence upon the first data.
 15. The method according toclaim 10, wherein either: the display forms part of a head mounteddevice comprising the first electronic device; or the display forms partof the head mounted device which also comprises an interface supportingcommunications to and from the first electronic device.
 16. The methodaccording to claim 10, wherein the image comprising the modified imagedata comprises modified obtained image data within which thepredetermined portion of the obtained image data has been digitallyreplaced with the modified image data; the predetermined portion of theobtained image data is modified in order to address a first vision lossof the user with respect to a predetermined portion of their visualfield of view; and the modified obtained image data comprises theobtained image data modified in dependence upon a second vision lossrelating to a general degradation in the user's visual field of view.17. The method according to claim 10, wherein at least one of: thesecond electronic device is at least one of a digital image acquisitiondevice, a source of video data being displayed upon another displaywithin the field of view of the user, and a third electronic deviceassociated with the user; and the predetermined portion of the obtainedimage data to be modified is determined in dependence upon data acquiredtracking the user's eye movement and the first data.
 18. The methodaccording to claim 10, wherein the predetermined portion of the obtainedimage data modified relates to a characteristic of a vision loss of theuser within a predetermined portion of the user's visual field of view;the characteristic of the vision loss of the user relates to at leastone of: an element the user's optical imaging system selected from thegroup comprising cornea, pupil, lens, iris, and vitreous gel; a neuralcomponent of the user's vision; an environmental condition; compensatingfor missing portions of the user's field of view; correcting a problemwith the user's eye tracking; improving mental focus; and removingperipheral distractions.
 19. The method according to claim 10, whereineach mode of operation of the plurality of modes of operation relates toa different task performed by the user; and the predetermined subset ofa plurality of image processing algorithms associated with each mode ofoperation of the plurality of modes of operation is established by theuser.
 20. A method of improving the visual function of a user comprisingthe steps of: configuring within a non-volatile non-transitory memoryassociated with an electronic processor of a first electronic device aplurality of sets of second data; each set of second data relating toconfiguring a mode of operation of a plurality of modes of operation andrelates to a predetermined subset of a plurality of image processingalgorithms applied by the electronic processor; obtaining image datafrom a second electronic device; modifying a predetermined portion ofthe obtained image data with the electronic processor to generatemodified image data established in dependence upon applying thepredetermined subset of the plurality of image processing algorithmsassociated with a selected mode of operation of the plurality of modesof operation to the predetermined portion of the obtained image data;projecting an image comprising the modified image data onto a user'sretina via a display; wherein the selected mode of operation isestablished by the electronic processor either: by receiving input dataestablished by the user via an input interface of a portable electronicdevice coupled to the first electronic device, the input data relatingto the selection by the user of a mode of operation of the plurality ofmodes of operation; or in dependence upon the visual field of view ofthe user.
 21. The method according to claim 20, wherein each mode ofoperation of the plurality of modes of operation is established independence upon determined characteristics of a vision loss of the user.22. The method according to claim 20, wherein each mode of operation ofthe plurality of modes of operation is established in dependence upon atleast one of: characterisation of an element the user's optical imagingsystem selected from the group comprising cornea, pupil, lens, iris, andvitreous gel; characterisation of a neural component of the user'svision; an environmental condition; characterisation of one or moremissing portions of the user's field of view; characterisation of aproblem with the user's eye tracking; improving mental focus of theuser; and removing peripheral distractions within the field of view ofthe user.